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input method, and an information Input sheet (209) 
which a user can easily operate without directly having 
to become aware ol haidwafe are proved. Ah informa- 
tion input sheet (209) includes a display area having an 
input information area (E) and an idemlf icatloh area (A) 
in which an identif ication code of One input information is 
provided. The identification code of the input informa- 
tion is provided in the display area as a target pattern 
having a rotation mode in which a code is defined by a 
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directional orientation of th'eliSgeti pa 
pickup unit (101) ' qeneiate^H^hrig: \v^i^i^. 4 ff^^,^ . 
imaging the information inpi^^^t^he imaginjaifrrfo^ 
mation includes imaging in^^«0n^ the^^icati^J^ 
area of the information Input sheet.. Based *ori this* 
imaging information of the identification area* the rota- 
tion mode of the target pattern, as recognized by an 
image recognition unit (102). is determined, the rota- 
tion mode specifies the ideri the infor- 
mation input sheet Information on the information input 
sheet is authenticated by authentidaiibn ^ pr6<»ssing 
means(i03). 
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Description 

RELATED APPLICATIONS 

s The present application claims foreign priority under 35 U.S.C. Section 1 i 9(a)-(d) based on Japanese Patent Appli- 
cation P08-31 5289 filed in the Japanese Patent Office on November 26, 1 996. 

BACKGROUND OF THE INVENTION 

w Field of the Invention 

The present invention relates to a method and apparatus for entering data and/or control information into an elec- 
tronic device. More particularly the present invention relates to a system and method for employing image recognition 
to enter information into a computer or other data and/or control system. 

is 

Description of the Related Art 

Various new multimedia devices that combine the functions of existing audio-visual (AV), information, and commu- 
nication devices have been proposed which enable input/output of multimedia information. Examples of such new 

20 devices include portable information terminals with communications capabilities, camcorders with communications 
capabilities, and personal computers with both communication capabilities and AV functions. In addition, systems 
designed to allow these new multimedia devices to communicate with each other via different types of networks are 
under development Some of the cfifferent types of networks that are being used for i nter-device communication include 
Ethernet focal area networks (LANs), Token Ring LANs. ATM wide area networks (WANs), wire communication net- 

25 works like public telephone networks, and wireless communication networks such as infrared communication systems, 
cellular telephone systems, and satellite communication systems. 

Many of these new multimedia devices use the same conventional interfaces for inputting information as the exist- 
ing AV, information, and communication devices. For example, both the conventional and new multimedia devices use 
keyboards, mice, touch panels, dedicated controllers and microphones. 

30 As depicted in Fig. 15A, Bar codes are another example of a conventional means used fw erftering information into 
multimedia systems. A bar cod© typically includes a pattern of lines of varying thicknesses which represent binary 
codes. They are typically used for the machine identification of goods and other articles, or to specify a target object. 
Bar codes can be one or two dimensional. In other words* bar codes are arranged so that they can be scanned by a bar 
code reader in either one or two different scanning directions. Fig. 1 SB depicts a two dimensional bar code. 

35 Unfortunately, operating conventional input interfaces is awkward and not suited to the human senses. This is not 
to say that existing devices themselves are not ergonomic, but rather that their method of operation is hot intuitive and 
they typically require s^ 

many kinds of adjustments and special comfort features but until the user is taught and then practices touch-typing, 
even the most ergonomic keyboard is very difficult to use efficiently. The same is true for mice, touch panels, dedicated 

40 controllers and microphones used in conventional multimecfa devices Such input devices simply do not suit the human 
senses. In other words, users who ere not accustomed to operating such devices do not intuitively understand the asso- 
ciated operating methods, and need special knowledge and training to become skilled in operating these devices. 

In addition to not being easily record or read by humans, systems that use bar codes suffer from the added problem 
that there are only a finite number of codes for a given bar pattern display area. The more different articles that need to 

as be identified by such a system, the more unique patterns are required. Eventually, either the number of bar code pat- 
terns has to be increased or the number of articles that can be identified must be limited. Increasing the number of pat- 
terns necessitates increasing the area of the bar pattern display on the article. This is not a preferred solution. For 
example, in the case of a product the surface area of the article is finite and usually used for displaying other informa- 
tion. Further, adding more unique patterns by adding additional bars, which typically each only add two bits worth of 

so information, increases the time required to recognize the bar code. 

Thus, it is an object of the present invention to provide an information input apparatus, an information input sheet, 
and an information input method, which can be easily, efficiently, and intuitively operated without the user having to 
learn how to use or even become fully aware of specialized hardware. 

It is a further object of the present invention to provide an information input apparatus, and an information input 

65 method, which enables a user to specify, associate, and reference information from among a large volume of data 
through a simple operation. 

It is a still further object of the present invention to provide an information input apparatus, an information input 
method, and an information input sheet which enable a significantly increased the number of unique identifiers per unit 
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area lor use in the identification of goods and articles. 
SUMMARY OF THE INVENTION 

5 The above and other objects of the invention are achieved by the present invention of an information input method, 

information input sheet, and information input apparatus. The information input method according to the present inven- 
tion includes the steps of appending a predefined target pattern having a rotation mode in which a code is defined by 
an orientation, to the surface of an object, and inputting code Jnforrnafion corresponding to the rotation mode of the rec- 
ognition target pattern by image recognition of the recognition target pattern. 

io According to the present invention, there is also provided an information input sheet including a display area having 
at least an input irrformation area and an identification area The input information area provides input infc^rmtion and 
the identification area provides an identification code for the input information. In the display area, the tdentrf "ication code 
of the input information is embodied as a recognition target pattern having a rotation mode in which a code is defined 
by an orientation characteristic of the target pattern. 

15 According to the present invention, there is also provided an information input apparatus that includes an informa- 
tion input sheet including a display area having at least an input information area in which input information is provided 
and an identification area in which an identif ication code of the input irrformation is provided. The identification code of 
the input irrformation is embodied in the display area as a recognition target pattern having a rotation mode in which a 
code is defined by the directional an orientation ot the target pattern. TTie irrformation i 

20 an image pickup unit for imaging the display area of the information input sheet; an image recognition unit for performing 
image recognition of the recognition target pattern using imaging irrformation of the identification area of the information 
input sheet imaged by the image pickup unit; and processing means for determining the rotation mode of the recogni- 
tion target pattern recognized by the image recognition unit so as to specify the identif information 
input sheet and authenticatfon irrformation from imaging the information input sheet: 

25 These and other features and advantages of the present invention will be understood upon consideration of the fol- 
lowing detailed description the inv^ 

BRIEF DESCRIPTION OF THE DRAWINGS 

4 so Figure 1 is a functional block diagram depicting an embodiment of an information input apparatus according to the 
present invention. 

Figure 2 is a block diagram depicting the structure of a multimedia system within which the information input appa- 
ratus of Fig. 1 is apjiKed. 

Rgure 3 is a btock diagram depicting an embedment of a hardware structure of a portion of the information input 
25 apparatus dfRgl • 

Figure 4 is a plan view illustration depicting an example arrangement of an embodiment of a input sheet of the infor- 
mation input apparatus of Fig . 1, 

Figure 5A is an illustration depicting an example of code value assignments of rotated icon codes that can be used 
on the input sheet of Fig; 4. 
40 Rgure 5B is an illustration depkTfing aiirst example ^ rotated w 

• . • 4. . . . ' . . .. ... i , 

Rgure 6 is a plan view illustration depicti an example embodiment of an input sheet of Fig. 4 for controlling an 
electronic d^k^ having co 

Rgure 7 is an illustration ducting an exemplary output display of an output unit of an embodiment of an irifbrma- 
45 tton input apparatus according to the present invention using the input sheet of Fig, 6. 

Rgure 8 is a flowchart depicting an embodiment of an identification processing method of recognizing a target with 
an image recognition unit of an embodiment of an information input a^aratus accord to the present invention. 

Rgure 9 is an illustration depicting the amount of shift detected during position correction by the image recognition 
unit of an embodiment of an information input apparatus according to the present invention. ; 
so Figure 10 is an illustration depicting the center of mass and a tircumscrfoed rectangle of each icon extracted by 
rotated icon extraction processing in an embodiment of ah identif ication processing method of recognizing a target 
according to the present invention. 

Rgure 11 fe an illustration depicting a part of a pre-defined rotated icon code group used in rotated icon code deter- 
mination processing in an embodiment of an identification processing method of recognizing a target according to the 
65 present invention. 

Rgure 12 is a flowchart depicting an embodiment of an inpuVselection irKiication processing method iri an embod- 
iment of an image recognition unit of an embodiment of an information input apparatus of the present invention. 

Rgure 13 is an illustration depicting an access indication input pattern giving an input/selection indication extracted 
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by input/selection indication extraction processing performed by an image recognition unit ol an embodiment of an infor- 
mation input apparatus of the present invention. 

Figure 14 is an illustration depicting a second example of rotated icons described on the input sheet of Fig. 4 
Figure 1 5 A is an illustration depicting different formats of one dimensional bar codes used with input systems of the 
5 prior art. 

Figure 15B is an illustration depicting a two dimensional bar code used with input systems of the prior art 
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

to Turning to Fig. 1 . a preferred ernbodimerrt of an information input apparatus 100 according to the present invention 
includes an image pickup unit 101, an image recognition unit 102, a control unit 103, a storage unit 104 and an output 
unit 1 0S. The control unit 103 is connected to an information distribution unit 1 06 via communication link 1 08 and to an 
external device 107 via communication line 109. Communication link 108 can be for example, a network, while commu- 
nication line 109 can be for example, a home bus. 

15 Fig. 2 depicts an application of the information input apparatus 100 of Fig. 1 . In other words, Fig. 2 illustrates the 
present invention as H could be used in, for example, a multimedia system. Referring to both Figs. 1 and 2, the image 
pickup unit 101 images a recognition target pattern appended onto an object surface, such as paper, and an access 
indication pattern made by a user with a finger or any other similar pointer. Specifically, the image pickup unit 101 is 
preferably embodied as sensitive digital camera such as a charge-coupled-device (CCD) video camera 201 as shown 

20 in Fig. 2 or a Quantum Well Infrared Photodetector (QWIP) video camel The CCD video camera 201 is preferably 
located on top of a viewer 205A as in the multimedia system of Fig. 2. The CCD video camera 201 is focused to capture 
images of an input sheet 209 set on a medium base 208. The medium base 208 is adjacent to the viewer 205A which 
is disposed in an upright position so that the CCD video camera 201 , located on top of the viewer 205 A, can be focused 
on input sheet 209 on the medium base 208. 

25 In such an embodiment, the input slieet 209 serves as an information selection medium which is segmented into 
different category areas. The input selection sheet 209 can be made, for example, of paper or a thin plastic sheet. Turn- 
ing to Fig. 4, an embodiment of an arrangement of the input sheet 209 of Fig. 2 is shown. The embodiment depicted in 
Fig. 4 illustrates an example input sheet 209 that includes five different category areas labeled A, B, C. D and E. Note 
that a particular category can include multiple distinct regions on the input sheet 209. 

30 In the first category area A of the input sheet 209, an authentication icon is provided that can be used for authenti- 
cating that a particular plastic sheet or piece of paper is an input sheet 209 that will be permitted to be used with the 
information input apparatus 10Q of the present invent ipa In other words, if the input sheet 209 includes an authentica- 
tion icon in category area A that the infomiatfon input apparatus 100 is able to authenticate, then the system will enable 
operation with that particular input sheet 209. Otherwise, the system will be disabled from operating with that particular 

3s input sheet 209. The combinatjon of an authentication icon located in a predefined category area is referred to as a rec- 
ognition target pattern. 

In the second category area B of a preferred embodiment of an input sheet 209 there is an identification icon (or 
icons) for associating a particular sheet with stored or accessible electronic information. The identification icon is also 
referred to as a recognition target pattern. In the third category area C, an input sheet 209 position correction mark is 

40 provided. In the fourth category area D, a graphic for a user to indicate a selection is provided. Finally, in the fifth cate- 
gory area E, some form of a listing of the information content of the system is provided. The structure of an input sheet 
209 and the arrangement of the gr^hics H provkJes is further desctfo^ 

The image recognition unit 102 of Fig. 1 performs image recognition on image ir^rmatton provided by me image 
pickupunit 101. In the embodiment of Fig.2,tneCCD video camera 201 generates image information from imagingthe 

4s input sheet 209. The image recognition unit 102 is embodied within a control box 202 in the multimedia system shown 
in Fig. 2. The control box 202 with the image recognition unit 1 02 receives the image information output from the CCD 
video camera 201. The image recognition unit 102 performs identification and authentication processing of the recog- 
nition target patterns contained within the image information received from the CCD video camera 201. Based on the 
identification and authentication processing, the image recognition unit 102 outputs an identification result and an 

so authentication result to the ^ control unit 103. 

As will be described below in detail, once an association between an input sheet 2Q9 and electronic information is 
made as a result of recognizing the recc^nrtion target pattern, the image recognition unit 102 proceeds to recognize an 
input/selection indication made by the user and then outputs a recognition result signal to the control unit 103. 

Along with the image recognition unit 1 02, the control unit 1 03 is also provided within the control box 202 of the mul- 

55 timedia system shown in Fig. 2. The control init 103 controls transmission and reception of information between the 
various other components. Based on the recognition result of the recognition target pattern by the image recognition 
unit 102. the control unit 103 determines whether the recognition target pattern is associated with information that is 
stored locally in the storage unit 104. If so. the control unit 103 accesses the associated locally stored information. 
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If the recognition target pattern is associated with irrformation that is not stored locally, the recognition result is 
transmitted to the information distribution unit 106. In this case, the associated irrformation is stored in the information 
distribution unit 106 or in an irrformation processing unit located on an external oomrruni(»lion network connected to 
the information distribution unit 106. The information distribution unit 106 accesses the associated remotely stored 
5 information. 

In either case, once the associated information is located, the control unit 103 transfers a copy of the associated . 
information into a temporary storage unit (not pictured) provided within the control unit 1 03. The associated information 
preferably includes several types of data. For example, information associated with an input sheet 209 preferably 
includes display information for providing the user feedback on the output unit 105, control information for controlling an 

io external device 107, link information to locate other related irformatk>n, and command configuration information for 
interpreting and using the input sheet 209. 

Once the control unit 103 has a copy of the associated information in the temporary storage unit authentication is 
performed for each input/selection indication image received from the image recognition unit 102 As the input/selection 
indication images are authenticated, a display screen on the output unit 105 is updated and control functions of the 

is external device 107 are performed. In other words, presentations iricB<^ing the 

tfons and information distribution results are displayed on the output unit 1 05 as they; are performed. 

The above-listed types of information that are associated with different input sheets 209, or different areas of an 
input sheet 209, will new be explained in further detail. The display information includes characters, images, layout 
information, sound information and the like. Based on this display information, the control unit 103 creates displays on 

20 the output unit 105 and controls sound generation. 

The control information includes control commands for controlling an external device 1 07 in response: to the user 
making an input/selection indication on an input sheet 209. The control unit 103 controls the externa! device 107 based 
on the control information associated with the input sheet 209. For example, if the external device 1 07 is a video cas- 
sette recorder (VCR), the control irrformation includes reproduction (play), stop, fast forward, rewind, and pause control 

25 information for controlling the operation of a VCR: When, for example, the user points to a graphic on the input sheet 
209 that represents the play control information, the image pickup unit 101 sends image information to the image rec- 
ognition unit 1 02 which, in response, recognizes the play input/selection indication image and sends a play command 
to the VCR. . •■■ ; • ' ■ 

The link information includes addresses for reading out other information when the user makes an input/selection 

so indication. The links can point to information stored in the storage unit 104, the information distribution unit 106, or an 
external information processing unit connected to the communication network. Based on the link information, the con- 
trol unit 103 accesses other additional information. For example, a Uniform Resource Locator (URL), as used dn the 
Internet, may be used as link information in the present invention. ■ 

The command configuration information includes information indicating which functions are to be performed when 

35 a graphic within the D or E category areas of the information input sheet 209 is selected by the user. The proofs of a 
user selecting a graphic on the input sheet 209 is also referred to herein as a user input/selection indication. The com- 
mand configuration information is in the form of a physical position on the input sheet 209 as sd e^ied 1^ the user and 
a next processing command corresponding to that selected position. For example, the command configuration informa- 
tion could be a position coordinate and a processing command for executing the above-descrcb«J control information 

40 or link information when the appropriate posrtkxi exxxdinate is selected by the user. 

In addition to the control unit 103 and the image recognition unit 102, the storage unit 104 is also proved^ within 
the control box 202 in the multimedia system shown in. Fig. 2. the storage unit 104 st information associated with 
the recognition target pattern The information stored in the storage unit 104 includes the information for controlling the 
external device 107. This information can be retrieved from an external information processing unit connected to the 

45 network via the information distribution unit 106 or. for example, from a remote information source via a satellite com- 
munications system; By locating the storage unit 104 within the control box 202, the traffic on the network may be 
reduced and the response time to user inpuVselectioh indications may be shortened. 

The output unit 105 preferably includes a display, a speaker, ami/or a printer. Thus, the output unit 105 can provide 
sound and a visual display of both the function of the operation selected by the user arid the information distribution 

so results. Fig. 7 is an exaririple of sample irrforrnatiori output to an output urtit 105 display the i leift scdG rof the 
display screen 700 there is a graphic 710 illustrating the function of the operation selected by the user. Note that VGR 
controls are represented and the play command graphic 720 (a tight pointing arrow) is darkened. This indicates the user 
selection of the play command. On the right side of the display screen 700 there is an image 730 output from the exter- 
nal device 107 which represents the information distribution results. Image outputs of various other devices under con- 

55 trol of the system may also be synthesized and outputted to the output unit 1 05 as shown in Fig. 7. The output unit 105 
of the multimedia system shown in Fig. 2 is embodied as a viewer 205A and a video scan converter 2058. 

The information distribution unit 106 can be connected to the control unit 103 via wire communication networks 
such as Ethernet LANs, Token Ring LANs, or ATM WANs. Alternatively, wireless communication networks such as infra- 
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red, PHS telephone, digital cellular telephone, or satellite communication systems can be employed. 

When the control unit 103 determines that the recognition result Irom the image recognition unit 102 requires infor- 
mation associated with a particular input sheet 209 that is not available locally in the storage unit 104, the information 
distribution unit 106 retrieves the associated information from an external information processing unit connected to the 

5 system via the communication network. In the multimedia system shown in Fig. 2, the irrtormation distribution unit 106 
is embodied as a programmed workstation 206. 

The external device 107 can be any number of different electronic devices that can be controlled by the control unit 
103 in response to the input/selection indication. For example, in Fig. 2, the external device 107 includes an AV control- 
ler device 207A, a VCR 208B. a modem 207C, a telephone set 207D. and a camera 207E. The external device 107 

w could also include many other types of electronic devices, such as. computers and game machines. The external device 

107 can be connected to the control unit 103 via wire communication lines or wireless communication systems. 

Fig. 3 depicts an exemplary hardware embodiment of the control box 202 of Fig. 2. The control box 202 of Fig. 2 
includes a central processing unit (CPU) 30 1 , an image processing unit 302, a camera interface 303, a display interlace 
304. a network interface 305. a home network interface 306, a read only memory (ROM) 307, a random access memory 
15 (RAM) 308, a hard disk drive (HDD) 309. All of the components of the control box 202 can be interconnected via a bus 
310. Alternatively, these components can be connected to each other via dedicated communications paths. For exam- 
ple, the camera interface 303 is shown connected directly to the Image processing unit 302 by a dedicated link. Like- 
wise, the CPU can have a second separate bus for connecting to the ROM 307 and/or RAM 308. 

The CPU 301 is adapted to control the control box 202 in accordance with a system program stored in the ROM 
20 307 or a program developed in the RAM 308. The CPU 301 provides some of the functions of the image recognition 
unit 102 and some of the functions of the contrd unit 103 depicted in Fig. 1. In other words, both the image recognition 
unit 102 and the control unit 103 (»n be in part ernbcxJied as a CPU 301. 

Programs and data necessary for the CPU 301 to perform various types of processing are stored in the ROM 307. 
The ROM 307 provides some of the functions of the control unit 103 of Fig. 1 . The RAM 308 is adapted to develop and 
25 temporarily store programs and data necessary for the CPU 301 to perform various types of processing. The RAM 308 
also provides some of the functions of the control unit 103 of Fig. 1 . In other words, the control unit 103 can be in part 
embodied as a ROM 307 and a RAM 308 together with a CPU 310. The HDD 309 provides the functions of the storage 
unit 104 of Fig. 1 and thus, the storage unit 104 can be embodied as a HDD 309. 

The image processing unit 302 receives image information from the CCD video camera 201 via the camera inter- 
so face 303. various types of image processing are performed by the image processing unit 302 such as image recogni- 
tion processing. The image processing unit 302 provides some of the functions of the image recognition unit 1 02 of Fig. 
1 . The camera interface 303 receives image information from the CCD video camera 201 and then converts the image 
information to a signal format compattole with the image processing unit 302. The camera interface 303 thai outputs 
the converted image information to the image processing unit 302. Thus, the image recognition unit 1 02 can be embod- 
35 ied in part as a camera interface 303 and an image processing unit 302 together with a CPU 301. 

The display interface 304 receives display data processed by the CPU 301 and the image processing unit 302. con- 
verts the display data to signals compatible with the viewer 205 A, and then outputs the converted signals to the viewer 
20SA The display interface 304 and the viewer 205A provide some of the functions of the output unit 105 of Fig. 1. 
Thus, the output unit 105 can be embodied in part as a display interface 304 and a viewer 205A 
40 The network interface 305 provides a connection to the workstation 206, thereby enabling access to an external 
network. The network interface 305 and the workstation 206 provide some of the functions of the communication link 

1 08 and the information distribution unit 106 pf Fig. 1. Thus, the communication link 108 and the information distribution 
unit 106 can be embodied in part as a network interface 305 and a workstation 206. 

The home network interface 306 provides a connection to the external device 107. The CPU 301 is thus able to 
45 control the external units 207A-207E via the home network interface 306. The home network interface 306 and the 
external units 207A-207E provide some of the functions of the communication line 109 and the external device 107 of 
Fig. 1 . Thus, the communication line 1 09 and the external device 1 07 can be embodied in part as a home network inter- 
face 306 and a plurality of external units 207A-207E such as VCRs, modems, video cameras, etc. 

The above described constituent components 301 -309 are connected to one another via a bus 310. Together, they 
so provide the various functions of the intormation inp^ apparatus as descrbed above with reference to Fig. 1. 

The input sheet 209 imaged lay the irtormation input apparatus as described with reference to Figs. 1 to 3 above is 
explained below in grater detail with reference to Fig. 4. As described above, the input sheet 209 is preferably seg- 
mented into the five category areas A, B. C, D and E. 

In the first category area A of the input sheet 209, an authentication icon is provided for authenticating that a par- 
es ticular plastic sheet or piece of pcper is an input sheet 209 that will be permitted to be used with the information input 
apparatus 100 of the present invention. As explained above with reference to Figs. 1, 2 and 3, the authentication icon 
is provided as a recognition target pattern. The authentication icon in the category area A is imaged by the CCD video 
camera 201 and compared with comparative icon information stored in the ROM 307 by the CPU 301 . When the CPU 
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301 is able 1o find a match between the authentication icon of the input sheet 209 and the comparative icon information 
stored in the ROM 307, operation proceeds to identification processing using the category area B. H no match exists, 
an error message is displayed on ^ 

In the second category area $i an tderrtrtication k» (or icons) lor ass^iatihg the 

5 information is provided as a recognition target pattern. The identification icon has a directionality and a predefined 
number of recognizably distincit driie^ in the Fig. 4 embodirnent of the input sheet 209, an icon 

resembling a key is used. The key shape has a narrow end and a wide end that gives it directionality and allows one to 
identily its orientation. Further, different orientations of the identification icon represerH dififerer^ valu codes or mean- 
ings. In the embodiment pictured in Fig; 4. each identification icon that serves as part of the recognition target pattern 

10 in the category area B, has eight distinct orientations. Thus, each identification icon can be oriented such that eight dis- 
tinct cede values can be represented bdsed spldy on the orientation of the icon. Fig 5A illustrates an example of eight 
distinct orientations for the key shaped icon with one of eight distinct code values assigned to each orientation. 

Through the use of just a few additional distinctly oriented icons together in a sequence, many more distinct codes 
can be represented. For example, using a sequence of four identification icons, each having eight distinct orientations, 

15 4096 different code values can be represented. The number of values that can be represented by four icons with eight 
orientations is computed as follows: 

(8 orientations) ^/(S orientations) ^*(8 orientations) lcon 3*(8 orientations) Jcoi>4 =8 4 «4096 values 

20 Using the example orientation value assignments depicted in Fig. 5A, the ordered sequence of identification icons 600 
shown in Fig. SB represents a code value of "2574". That is to say, the first key icon 510 represents "2", the second key 
icon 520 represents "5". the third key icon 530 represents "7", and the fourth key icon 540 represents "4". 

In the third category area C, a position correction marks are provided. The position correction marks are used for 
performing position correction of the image picked up by the CCD video camera 201. 

25 In the fourth category area D, a graphics are provided for the user to perform aselection indication. By selecting a 
graphic in area D. the user can select information which is displayed on the viewer 205A but is not displayed in the cat- 
egory area E of the input sheet 209. That is, the category area D is used to make a selection indication whenever the 
selection information provided in the category area E of the input sheet 209 differs from the selection information dis- 
played on the viewer 205A. This allows the information input apparatus 100 to dynamically add input selection options 

so not present in category area E, via the viewer 205A. 

For example, as shown in Fig. 4, when up; down; left and right arrowhead graphics aro area 
D, the user can choose an arbitrary arrowhead graphic in category area D to perform the shifting of a cursor and/or the 
determination of a menu selection on a menu screen displayed on the viewer 205A. Although the graphics provided in 
the category area D can be embodied sis an up; ctawh, left, and/or rigW 8/rcw figures tp cursor rnovement func- 

36 tions, the graphics are not necessarily limited to such arrowhead shaped figures and the functions are hot necessarily 
limited to such curs<x rnovem^m fonctibns: 

For example, if the input sheet is to be used as a oommon medium for controlling both a VCR and an audio system, 
graphics for controlling furictiom com to both external devices are provided. VCBs and audo systems frequently 
have functions in common such as, playback, stop, and record functions. Thus; by associating graphics in the category 

40 area D with command conf iguration information which includes embedded process ng eb™ 

tion. execution of the embedded processing commands is enabled. In the aJDOve-descrfoed example, me embedded 
processing commands include: upward, downward, leftward, and rightward cursor shift commands, and play, stop, and 
record commands. But for example,: for a compact disc (CD) player, similar graphics could be used for commands such 
as forward within track, forward to next track, reverse within track, reverse to previous track, eject, stop, and change 

45 information display mode. Recall that command configuration information is copied into the temporary storage unit 
within the control unit 103 once an information sheet 209 is authenticated and identified. 

The content of the substantive information available via the information input apparatus 100 is provided in the fifth 
category area E. This contents listing can be displayed in many different ways. For example, it canbe displayed in the 
form of a table that enumerates general topics arranged in a subject matter based order, or as an image map analogous 

so to a hyper-text transmission protocol (HTTP) image map used on World Wide Wel> (WWW) pages on the Internet, 
and/or as a detailed index that itemizes each an every bit of substantive irifctrrnalidn available in a aJphabeticai order. 
By viewing the information described in this category area E, the user can learn what information is accessible with the 
particular input sheet 209 currerrfy imaged by the CCD video camera 201 . 

In addition, once a particular input sheet 209 is recognized by the information input apparatus 100 shown in Fig. 1, 

66 the user may select from the x^ertfs informtfon provided in this category area E by simply poi nting to the desired item 
of intorriation within the contents information. The step of pointing provides an indication selection image pattern that 
can be recognized by the interface system 100. The indication of a particular selection is then provided to the control 
unit 103 which then can perform some predetermined processing. 
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As an example, consider the input sheet 209 of Fig. 6. The category area E includes a graphic image map of VCR 
control buttons. The graphics 600 that represent the six different control buttons provided an intuitive set of controls that 
correspond to the VCR functions of play 610. stop 620. pause 630, rewind 640, fast forward 650, and eject 660. This 
example arrangement allows the user to easily operate the VCR using the six control buttons. Although the information 

5 contents described in the category area E has been explained using the control buttons for a VCR, one skilled in the art 
would realize that many different devices can be controlled using an appropriate input sheet 209 and, in particular, that 
the information contents are not necessarily limited to such VCR control buttons. 

As a further example, the contents information in the category area E could be in the form of a series of menus. 
Thus, in response to the user pointing to a particular menu in the category area E using a finger, a subsequent display 

io of menu items that correspond to the selected menu is displayed on the viewer 205A. Thus, by associating menus in 
the category area E with link information and command conf iguratiqn information which includes embedded processing 
commands and position information . the display of link information and execution of the embedded processing com- 
mands is enabled. In this example a subsequent list of menu items are displayed on the viewer 205A in response to the 
user pointing to the location of the related initial menu on the input sheet 209. Once again, associated link information 

is and command configuration information are copied into the temporary storage unit within the control unit 103 once an 
information sheet 209 is authenticated and identified. 

In a multimedia system structured as described above, the image recognition unit 102 carries out identification 
processing for identifying the type of input sheet 209 being used. This happens once the input sheet 209 has been 
authenticated by authentication processing ol the image of the input sheet 209 received from the CCD video camera 

20 201. 

The authentication processing is carried out by first performing pre-processing, such as, elimination of noise, vari- 
able density processing, and adjustment of threshold valua Next position correction is performed. The system then pro- 
ceeds to extract the center of mass and contour of the authentication icon used as the recognition target pattern in the 
category area A at the upper center of the input sheet 209. Finally, the system attempts to match the image information 

25 with a stored the authentication icon pattern. 

Next identification processing takes place. For example, as shown in the f lowchart of Fig. 8, pre-processing (S801) 
is first performed, such as, elimination of noise, variable density processing, and adjustment of a threshold value, with 
respect to the image information obtained as imaging output of the CCD video camera 201 focused on the input sheet 
209. Then, position correction processing (S802) is performed by first detecting and measuring a positional shift based 

30 on the image information of the position correction mark provided irrthe category area C preferably located at the four 
comers of the input sheet 209. The position correction processing (S802) then performs a transformation of the coordi- 
nate system corresponding to the amount of the positional stuff. Next, extraction processing (S8Q3) of the identification 
icon provided as a recognition target pattern in the category area 8 of the input sheet 209 is carried out and determi- 
nation processing (SB04) of a rotated icon code is performed. 

36 In the position correction processing (S802), the positional shift from a reference position is detected based on the 
image information obtained as imaging output of the CCD video camera 201 focused on the category area C of the 
input sheet 209. As shown in Rg. 9, the system overlays a correction pattern on the position correction marks and 
detects the direction and magnitude of any position error. For example, coincidence 900 occurs if the sheet is properly 
aligned, shift to upper left 910 occurs if the input sheet 209 is too high and off to the left, sniff to upper right 920 occurs 

40 if the input sheet 209 is too high and off to the right Unless the coincidence 900 pattern is detected, position correction 
is then carried out by mathematically transforming the coordinate system of the image recognition in accordance with 
the direction and magnitude of the positional shift: 

In the extraction processing of the identification icon (S803), the center of mass of variable density level and a cir- 
cumscribed rectangle are found for each icon from the image information obtained as imaging output of the CCD video 

45 camera 201 focused on the category area B of the input sheet 209. as shown in Rg. 10. In Fig. 10, the center of mass 
of each icon as determined by the identification icon extraction processing (S803) is indicated by a white spot and the 
circumscribed rectangle is indicated by a broken line. 

In the rotated icon code determination processing (S804), one of eight different orientations of the icon are deter- 
mined by referring to a direction characteristic table of the rotated icon. The direction characteristic table defines the 

so eight cGfferent orientations in terms of two parameters; the canter of mass and the aspect ratio of the circumscribed rec- 
tangle found in the identificaiton icon extraction processing (S603y The direction characteristic table is expressed as 
shown below in Table 1. It is predetermined based on the conditions indicated by characteristics of each icon orienta- 
tion. For example, the second row of the table is read as: "if the aspect ratio of the circumscribed rectangle is 2.5 or 
greater and the position of the center of mass is in the upper half of the circumscribed rectangle, the rotated icon is ori- 

65 ented downward." Based on the assignment of the icon's orientation with a code value as shown in Fig. 5A, the rotated 
icon is converted to a code. This processing is repeated for each of the four icons so as to determine a sequence of four 
codes. 
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TABLE 1 



Orientation of Icon 


Characteristics 


upward 


aspect ratio 52, center of mass located in lower half of rectangle 


downward 


aspect raiio o^, cemer or mass locaieo in upper nan or. reciangie 


rightward 


aspect ratio 2:5, center of mass located in left half of rectangle 


leftward 


aspect ratio 2:5. center of mass located in right half of rectangle 


upper rightward 


aspect ratio 1:1, c^erot mass located iriicwer left area\^^ 


lower rightward 


aspect ratio 1:1. cen^ area of rectajigle 


upper leftward 


aspect ratio 1 : 1 ,• carter of mass located in tower right area of rectangle 


lower leftward 


aspect ratio 1 :1, center of mass located in upper right area of rectangle 



Next, the rotated icon code sequence found from the four rotated icons is compared with a pre-defined rotated icon 
code group as shown in Fig. 1 1 . The best match between the rotated icon code sequence arid one of the pre-defined 
20 code groups is selected as the rotated icon code of the input sheet 209. The best match is determined by finding the 
pre-defined rotated icon code group with the smallest sum of differences between the respective parts of the pre- 
defined rotated icon code group and the rotated icon code sequence H the sum of differences of all the pre-defined 
rotated icon code groups exceeds a threshold value, the recognition target is not recognized as krwwn code and is 
judged as unrecognizable. 

25 As described above, the control unit 1 03 determines whether or hot an association between the recognition result 
of the recognition target by the image recognition unit 1 02 and some of tiie information stored in the storage unit 104 
can be made. If an association can be made, that information is accessed. That is, rf association with information stored 
in the apparatus itself can be made, the associated information is immediately accessed. If there is no information asso- 
ciated with the recognition result from the image recognition unit 102 that is currently stored in the storage unit 104. the 
30 recognition result is transmitted to the informatfon d^ 

In this case, the associated information is stored in the information distribution unit 106 or the information process- 
ing unit existing in the external communication network connected to the information distribution unit 106. Thus, after 
the information associated with the recognition result of the recognition target by the image recognition unit 102 is 
located by the information distribution unit 106. the control unit 103 reads the associated inforrnation into the temporary 
35 storage unit provided in the control unit 103. 

Thus, when identification of an input sheet 209 and determination of its type is completed, processing is performed 
based on the information read into the temporary storage unit provided in the control unit 103. This all occurs in 
response to the input/selection irxlication from the image recognition unit 102. 

Fig. 12 illustrates an embodiment of the steps us<ed in the inpuVs^edion indication processing by the irnage rec- 
40 ognition unit 102. Pre-processing (SI 201) is first performed. This includes functions such as, elimination of noise, var- 
iable density processing, and adjustment of threshold value. The pre-processing (S1 201) is performed on the image 
information obtained as imaging output of the CCD video camera 201 during the inpuVselection indication on the input 
sheet 209 as made by the user. 

Next position correction processing (S1202) is begun by detecting a positional shift as shown in Fig; 9 based on 
as the image information of the position correction mark provided in the category area G at the four corners of the input 
sheet 209. Position correction processing (S1 202) proceeds by computing the mathematical transformation of the coor- 
dinate system corresponding to the rrtagnitude and direction of the positional shift as described above with reference to 
Fig. 9. As a result, position-corrected binary image information is obtained. 

In the next step, extraction processing (S1 203) of the irtput/selection indication is performed, and then determina- 
50 tkxi processing (S1 204) of the inpiitfselect^ 

In the input/selection indication extraction processing (S1203), a pre-stored prijginai image of the input sheet 209 is 
compared to a variable-density image where the input/selection indication is detected. The position-corrected variable- 
density image information which, for example, could be a pattern indicated by a finger as shown in Fig. 13, is extracted 
as a differential image from the original image of the input sheet 209. That is, an access indication input pattern is deter- 
65 mined for specifying the in put/selection indication. An indication area includes the entire extracted differential image. 
Thus, a circumscribed rectangle and a pixel distribution within the indication area can be determined. 

Next, in the input/selection indication determination processing (S1204). the direction of entry of the pointer into the 
indication area is determined from the pixel density distribution on each edge of the circumscribed rectangle. Image 
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scanning within the rectangle is carried out relative to the entry direction and thereby the end point position of the 
pointer is determined. In the example embodiment of Fig. 1 1, the entry direction of the pointer is found by confirming 
whether a large number of pixels of high density are distributed on a particular edge of the indication area. The category 
area E is divided into a plurality of blocks of known area and the image scanning is carried out by scanning for pixels 
5 within these blocks based on the pointer entry direction and the corresponding scanning pattern specified in Table 2. 
Thus, by scanning only the indication area of the image which is divided in to blocks, high-speed detection of the pointer 
end point within a limited scanning area is enabled. 



TABLE 2 



ENTRY DIRECTION 


SCANNING PATTERN 


entry from right edge 


scanning of left end blocks in rectangle from upper edge 


entry from left edge 


scanning of right end block in rectangle from upper edge 


entry from upper edge 


scanning of lower end blocks in rectangle from (eft edge 


entry from lower edge 


scanning of upper end blocks in rectangle from left edge 


entry from upper right edge 


scanning of lower left end blocks in rectangle from left edge 


entry from lower right edge 


scanning of upper left end blocks in rectangle from left edge 


entry from upper left edge 


scanning of lower right end blocks in rectangle from left edge 


entry from lower left edge 


scanning of upper right end blocks in rectangle from left edge 



25 The detected end point position is converted to corresponding c^rrandfirrfcrma^^ transmitted to the 

control unit 103. in the example shown in Fig. 13, the end point position of thefin^er is converted to 
area number "6\ In Fig. 13, the area indicated by cross hatching is a block within the category area E corresponding to 
the area number "6". 

Finally, the control unit 1 03 executes processing corresponding to the area number "6" which is defined in the com- 
$c mand configuration information contained in the retrieved associated information of the identified input sheet 209 stored 
in the temporary storage unit of the control unit 103. 

Therefore, in the information input apparatus 100, authentication processing is performed by image recognition of 
the recognition target pattern provided on an input sheet 209. That is, a static medium can be used as the recognition 
target and an association between the recognition target and corresponding information can be made. The access indi- 
35 cation input specifying the associated information is detected by image recognition of the access indication input pat- 
tern. Thus, dynamic access of information can be carried out using a recognition target displayed on a static medium. 

The recognition target pattern is not limited to a key-shaped pattern as in the above described embodiment. It can 
be any distinct pattern with some directionality. Therefore, by combining the shape and the orientation of other recog- 
nition patterns, for example, as shown in Fig. 14, an infinite number of different types of input sheets 209 can be defined. 
40 Also, after recognition of the input sheet 209, the input/selection indication may be performed by merely pointing 
with a linger to the desired input/selection indication contents of the input sheet 209, Thus, the information input/out- 
put/selection operation may be easily carried out without directly becoming aware of the hardware, and the input/selec- 
tion indication contents can be associated with specified information from among a very large volume of data. 

In addition, in the above-described embodiment the center of mass of a : --yana|^^.0n^l'evel' and the circum- 
45 scribed rectangle of an icon's image are used for recognition of the rotated icon code. However, the orientation with 
respect to a prepared template image may be found by matching processing 

In an information input method according lip the present invention, a recognition target pattern having a rotation 
mode in which a code is defined by an orientation is appended onto an object surtax and tn response, code informa- 
tion corresponding to the rotation mode of the recognition target pattern is inputted by imajge recognition of the recoo> 
bo nrtion target pattern. Thus, many different of code values, corresjponding to the Afferent rotation modes can be 
generated for a given recognition target pattern Further^ the number of code values can be significantly increased. 

An information input sheet according to the present invention has a display area having at least an area in for input 
information and an identification area for an identification code of the input information: In the display area, the identifi- 
cation code of the input information is embodied as a recognition target pattern having a rotation mode in which a code 
65 is defined by an orientation. Thus, many iderrtrf teation code values are available for a given recognition target pattern, 
so that the number of unique identifiers per unit area is significantly increased. 

Also, in an information input sheet accord ng to the present invention, because a position correction area is pro- 
vided at a plurality of locations in the display area, and each position correction area including a position correction 
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mark, information for performing position correction during the image recognition processing can be determined from 
any of the position correction marks. 

Also, in an information input device according to the present invention, the use of an input sheet with a plurality of 
position correction marks insures that the recbgnitibn target pattern of a particular input is accurately perceived by the 
5 information input apparatus. Thus, the security of the recognition process and overall system is maintained by prevent- 
ing a recognition target pattern from being misperceh/ed as the recognition target pattern of a different sheet as the 
result of a position error. 

An information input apparatus according to the present invention includes an ihformatioh input sheet The infor- 
mation input sheet includes a display area having at least an input information area in which input information is pro- 

10 vided. The display area of the information input sheet also includes at least one identification area in which an 
identification code of the input information is provided. The identification code of the input information is embodied in 
the display area as a recognition target pattern. The target pattern includes a rotation mode characteristic which is 
defined by the directional orientation of the terget pattern. Imaging information is plained by imaging the information 
input sheet with an image pickup unit. The recognition target pattern on the irrforrnation input sheet is recognized by an 

15 image recognition unit processing tie imaging information output by the irnei^e pickup unit. Based on the imaging infor- 
mation of the identif ication area* the relation mode of the recognition target pattern is determined. As mentioned above, 
the rotation mode of the target pattern specifies an identification code of the information input sheet. Once this deter- 
mination has been made and the information input sheet has been identified, information entered via the information 
input sheet is authenticated by authentication processing means. Thus, information may be securely entered using an 

20 identified information input sheet. Also, many different identification code values, corresponding to the different rotation 
modes, can be generated using just one recognition target pattern on the information input sheet. Thereby, number of 
identification codes available for identifying different objects are significantly increased. 

Thus, the present invention provides an information input apparatus, an information input method, and an informa- 
tion input sheet which the user can easily operate without directly becoming aware of hardware. In addition, the present 

25 invention enables direct association of a small, simple interface with particular information contained within a large vol- 
ume of data. By simple operations the user is able to directly access the d^red information. The present invention also 
enables a significant increase in the nuniber of unique identifiers per unit area for use inthe identification of goods and 
articles. 

Various other modifications and alterations in the structure and method of operation of this invention will be appar- 
30 eht to those skilled in the art wrthbut darting scope and spirit of the invention. Although the invention has 

been descrfoed in connection with specific preferred embodiments, it should be understood that the invention as 
claimed should not be unduly limited to such specific embodiments. It is intended that the following claims define the 
scope of the present invention and that strudiiresr^ methods within the scope of these claims and their equivalents 
be covered thereby. 

35 

Claims 

1- An information input method comprising the steps of: 

40 appending a target pattern to an object surface, the target pattern having a rotation mode wherein a code is 

defined by an orientation of the target pattern; and 

inputting code information corresponding to the rotation mode of the target pattern by image recognition of the 
target pattern. 

45 2. The method of claim 1 , wherein the orientation of the target pattern is determined relative to a position correction 
mark appended to the object surfaice. 

3. An information input sheet comprising: 

so . ,\ " a. diqpiflv/larcwi- i^C^;!iAid^i w'! ! ! x 

an input information^ 

an idwtif'ratfoh area (A) in whkfr provided, 

wherein the identif k^tion code of the input inforrnation providcKJ in the identification area (A) of the dis- 
play area (209) includes a target pattern having a rotation mode in which a code is defined by an orientation of 
65 ■ the target pattera 

4. The information input sheet of claim 3, wherein the display area (209) further includes one or more position correc- 
tion areas distributed over the display area, each position correction area including a position correction mark. 
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5. An information input apparatus oompris ng: 

an irrformation input sheet (209) including a display area having an input information area (E) in which input 
irrformation is provided and an identification area (A) in which an identification code of the input information is 
5 provided, and wherein the identification code of the input information includes a target pattern having a rotation 

mode in which a code is defined by an orientation of the target pattern; 

an image pickup unit (101) for imaging the display area of the information input sheet and generating corre- 
sponding imaging information; 

an image recognition unit (103) lor performing image recognition on the target pattern using the imaging infor- 
w mation from the image pickup unit wherein the imaging information includes an image of the identification area 

of the display area of the information input sheet; and 

processing means (103) for determining the rotation mode of the target pattern as recognized by the image 
recognition unit so as to determine the identification code of the information input sheet and authentication 
information included on the information input sheet. 

15 

6- The apparatus of claim 5, wherein the information input sheet includes one or more position correction ar^as dis- 
tributed over the input sheet and each position correction area includes a position correction mark, and 

wherein the image recognition unit (102) includes position correcting processing of the target pattern using 
the imaging information from the image pickup unit wherein the imaging information includes an image of the post- 
20 tion correction areas of the display area of the information input sheet 
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